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Abstract 

A stationary state replica analysis for a dual neural network model 
that interpolates between a fully recurrent symmetric attractor net- 
work and a strictly feed-forward layered network, studied by Coolen 
and Viana, is extended in this work to account for finite dilution of 
the recurrent Hcbbian interactions between binary Ising units within 
each layer. Gradual dilution is found to suppress part of the phase 
transitions that arise from the competition between recurrent and feed- 
forward operation modes of the network. Despite that, a long chain of 
layers still exhibits a relatively good performance under finite dilution 
for a balanced ratio between inter-layer and intra-layer interactions. 

Pacs: 87.10.+e, 64.60.Cn, 07.05.Mh 



1 Introduction 

It is now well established that biological and artificial neural networks propa- 
gate information by means of synaptic interactions which may be symmetric 
or non-symmetric. The latter appear in biological networks, while either of 
both have been used in artificial networks. 
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There are two main classes of neural network models for associative 
information processing. One is that of recurrent networks with feed-back 
of either symmetric or non-symmetric synapses and the other one is 
that of layered feed- forward networks, with no feed-back loops, in which the 
synapses are intrinsically non-symmetric Models of the first class 

can be solved by means of equilibrium statistical mechanics in the case of 
symmetric synapses [3101, due to the presence of detailed balance, whereas 
one has to resort to a dynamical procedure in the search for stationary states 
in the case of non-symmetric synapses On the other hand, feed-forward 
networks are already dynamical systems which may reach stationary states 
in place of true equilibrium states. 

A novel dual model of binary neurons which combines recurrent and 
feed-forward processing, with symmetric synapses in the recurrent part, has 
been proposed some time ago by Coolen and Viana JJ, and the model has 
been generalized recently |12j-|14j. The network model consists of layers of 
fully connected neurons in a recurrent architecture that receive inputs from 
the previous layer (except the first one) and pass the processed information 
by means of feed-forward synaptic interactions to the next layer. The latter 
processes further the information and feeds, in turn, the next layer. Every 
unit in each layer is assumed to be connected to every other unit in that 
layer as well as to every unit in the next and in the previous layer but 
the information goes only from one layer to the next one (the feed-forward 
nature of the model). It is important to note that each unit in every layer 
participates of both the recurrent and the feed-forward process. Thus, a 
single updating procedure (usually, either sequential or parallel) has to be 
applied to every unit in each layer. 

The dual model becomes solvable by means of equilibrium statistical 
mechanics when all layers reach a stationary state and that may eventually 
be the case in the zero-temperature limit. In the case of finite temperature, 
thermal fluctuations destroy the Boltzmann form of the stationary states in 
the layers and one has to go over to a dynamical approach, which has not 
been done apparently so far. Actually, the model has been solved working 
out the free energy and the equations for the order parameters for finite 
temperature, assuming a Boltzmann form based on an underlying sequen- 
tial updating dynamics, and letting the temperature go to zero after the 
thermodynamic limit. The close agreement between theory and numerical 
simulations justified the procedure and the order of these limits despite the 
absence of detailed balance fH] . 

A feed-forward chain of recurrent networks is interesting as a model of 
dual information processing in which the feed-forward mode transmits the 



2 



outcome for the stationary overlaps within a layer to the next one. So far, 
only the case of a fully connected network has been considered, and one 
may ask how does the performance of the model change if the constraint 
of full connectivity between the units within each layer is relaxed by means 
of synaptic dilution keeping the full connectivity between units in adjacent 
layers. Synaptic dilution reduces the outcome from a layer and it could be 
that the cumulative effect in a long chain of layers would be a vanishingly 
small storage capacity even for a reduced amount of dilution. 

This issue is of interest in statistical mechanics and also for the informa- 
tion processing in biological networks, in particular for the understanding 
of the role played by the CA^ region of the hippocampus in the primate 
brain where the mean connection between units (our parameter c below) 
in the latter is of the order of 0.1 — 0.01 |151 116j . which may be considered 
as a finite dilution. Finite dilution may also be of interest if one thinks of 
defects in the connectivity within layers in artificial neural networks. De- 
tailed studies of the dynamics and the equilibrium states that describe the 
performance have been done in recent works that deal with finite dilution in 
various neural network models on diff'erent architectures jl7j-|21j following 
earlier works j22j-j24j. 

The main purpose of the present paper is to study the stationary states 
and obtain the phase diagrams that describe the performance of the dual 
model with finite dilution of the synaptic interactions within layers. In con- 
sistency with the procedure, we first take into account the disorder due to 
dilution for non-zero temperature, solve the model in the thermodynamic 
limit and then let the temperature go to zero. A second, minor purpose, is 
to indicate the necessary steps that yield the recursion relation for the pa- 
rameter that accounts for the input from the non-condensed overlaps which 
follows an unconventional derivation. It will be shown that, in addition to 
the well-known effect of finite dilution in replacing the intra-layer synaptic 
connection by an effective interaction that is the superposition of a Hebbian 
term for a fully connected network and a Gaussian noise, the only further 
contribution of the dilution on that parameter is an overall factor which can 
be absorbed by a simple rescaling in the case of dense networks. A further 
interest in the procedure to deal with the effects of synaptic dilution is that 
it may be extended to study the influence of general non-linear synapses 

The outline of the paper is the following. In Sec. 2 we review the model 
and introduce the alterations due to finite dilution. In Sec. 3 we solve the 
model to obtain the replica symmetric free energy near saturation and derive 
the recursion relations that describe the evolution of the parameters of the 
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network from one layer to the next. The phase transitions and some features 
of the performance are obtained and discussed in Sec. 4 and we end with 
concluding remarks in Sec. 5. 



2 The model 

The network model consists of L layers with N binary Ising units (neurons) 
on each layer I in a microscopic state <t' = {cj]^, . . . , cr^}, in which each 
a\ = ±1. The state +1 represents a firing neuron and the state —la 
neuron at rest. The microscopic dynamics of the network is assumed to be 
a Glauber sequential stochastic alignment of each neuron a\ to a local field 
h\ with probability 

Prob(a^ ^ -a\) = ^[1 - tanh(/3a|/i^)] , (1) 

where 

N N 

}iicT\a^-') = Y.Jl,a] + Y^K\^a^r^ , (2) 
j=i i=i 

is due to the states of other (see below) neurons <t' on the same layer and 
of neurons <t'~^ on the previous layer, as shown schematically in Fig.l. At 
each time step the neuron to be updated is taken at random from the set 
{<t'}. The parameter f3 = controls the synaptic noise such that the 
dynamics of the network becomes deterministic when T — > 0. 




Figure 1: The model. Two consecutive layers in a chain of length L with neu- 
ron states <t' and symmetrically diluted recurrent interactions {Jlj^ within 
layer I and feed-forward interactions {K[-} from the previous layer. 
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A macroscopic set of p = aN independent and identically distributed 
random patterns = {^f'', . . . , /i = 1, . . . ,p, are stored on the sites 
of each layer in the learning stage following a Hebbian rule 

4 = ^(i-^^.)i:efer ; i^i, = ^Eef^r^ (3) 

where each = ±1, with probability ^, is the component of the pattern 
^ on neuron i of layer I. Thus, a = p/N is the storage ratio of patterns 
per site of the Hopfield model PP, whether the sites are connected or not 
within a layer, since they are always connected to the sites of the adjacent 
layers. The parameters ( Jq, J) control the relative strength of the recurrent 
and feed-forward interactions, respectively, and they are the same for all 
layers, except on the first layer where = 0, for all and where one 

may distinguish between a free relaxation and a clamped operation defined 
below The recurrent part of the synaptic interaction is symmetrically 
diluted by means of the set of identically distributed random variables {cjj}, 
with Cij = Cji, such that Cij = 1 with probability c and zero with proba- 
bility 1 — c. The average number of connected neurons in each layer, the 
so-called connectivity, is cN and we focus our attention in this work on 
dense networks in which c = 0(1) implying that cN S> 1 within each layer 
|22j . When c = 1 the network is fully connected and letting c go to zero, 
after the thermodynamic limit — > oo, one has the network with strongly 
(symmetrical) diluted layers. One should expect that the synaptic dilution 
within the layers will have effects on the feed-forward information processing 
of the network, as long as Jq 7^ 0, and it is this that we want to study here. 

When J = the model reduces to a set of L decoupled symmetrically 
diluted attractor Hopfield-like networks which can be solved by means of 
equilibrium statistical mechanics using the replica method j22| I2,S[ I24j . In 
that case, the intra-layer synaptic interaction becomes an effective interac- 
tion which is the sum of a Hebbian term and a Gaussian noise that will be 
determined below. In the case of full connectivity (c = 1) the noise term 
vanishes and the critical storage ratio is that of the usual Hopfield model, 
ac ~ 0.138. 

In contrast, when Jq = 0, we have a purely feed- forward network with 
full inter-layer synaptic interactions between neurons on consecutive layers 
which can be solved exactly for all a, including the saturation of stored 
patterns, in the large N limit by means of a signal-to-noise analysis which is 
a truncated version of a neurodynamic approach developed some time ago 
j25| I26j . In that case, the local fields follow a Gaussian distribution allowing 
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for the derivation of exact recursion relations for the order parameters in 
the hmit of iV — > co [7j with a critical storage capacity ac ~ 0.269. We 
expect this capacity, for Jq = 0, to remain the same under exclusive synaptic 
dilution within the layers but there should be a gradual change of ac with 
increasing Jq and an eventual maximum ac for an intermediate situation 
between feed- forward and recurrent operation, as in the case of the fully 
connected network (c = 1) where ac ~ 0.317 

Coming back to the input at the first layer, one may have either a free re- 
laxation from a specific initialization as a recall cue or a so-called "clamped" 
operation where the recall cue is given by an externally specified but sta- 
tionary random state vector on that layer. We will discuss the effects of 
dilution on both modes. 

Due to the feed-forward nature of the interactions between consecutive 
layers the network will attain a stationary state at T = 0. This will be the 
case for any layer / when the input from layer / — 1 becomes stationary. The 
equilibrium states of the network may then be associated with the stationary 
states of the dynamics with a probability distribution given by a Boltzmann 
form p(<t') ~ e^-P^i^ ) ^ for any layer, with the Hamiltonian 

H{a^) = -\Y.jl^a\a]-Y^K\^aWr^ , (4) 

which enables one to solve the model first for finite T and then taking the 
T — > limit, as in previous work 

To study the performance of the network we introduce the macroscopic 
overlaps between the state and the stored patterns on that layer, 

i 

and assume, as usual, that in equilibrium only a finite number k of the pat- 
terns is condensed with overlap of 0(1), for = 1, . . . ,k, and the remaining 
p — k overlaps are assumed to be macroscopically small of 0(l/\/iV). The 
local field /i-(<t', <t'~^) will not only have contributions from these overlaps 
but also from the stationary overlaps of the previous layer I — 1, due to the 
second term in Eq.(2), 

i 

with similarly defined condensed and non-condensed components. 
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3 Free energy and stationary states 

We consider the partition function 

Z = J]e-^^('^) , (7) 

with H{cr) given by Eq.(4), and calculate the free energy treating, as usual, 
the non-condensed patterns fj, > k as quenched disorder over which we aver- 
age the logarithm of the partition function by means of the replica method 
to obtain the free energy / per site as 

-/3/=lim hm ^{{[Z^D^att-l} . (8) 

Here, {[Z''^]c) patt is the average over the quenched disorder, due to the pat- 
terns, of the connected replicated partition function [Z'^]c obtained by taking 
the average over the random symmetrical dilution. 

With the dilution only of synapses within the layers, we may write 

{<Ti}...{cr"} i<j 

where a (and /?, below) denotes the replica index, whenever it appears as 
a subindex, and {cr"} is the set of state vectors in the replica n. Using 
the now standard procedure to perform the configurational average over the 
symmetric dilution in the case of weak synapses J^j = 0{^y^) in the large 
N limit one finds 



a/3 i 

--p{^tE<iM} ■ (10) 



AcN 

i<j OL af3 

P 



Thus, the effect of the dilution amounts to the replacement of the intralayer 
synapses J^' • by the effective interaction |22l [J^ 



4e// = |E^r'^r'+^'. ' (11) 

for i 7^ j which is the sum of a Hebbian term for a fully connected recurrent 
network and a Gaussian noise of mean zero and variance 

= mi?]c = Jla{l - c)/c . (12) 
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Coming back to the averaged partition function we obtain 



4ciV 

{<Ti}...{cr"} al3 I 



exp{/3iV^^ [^m2 K^L) + Jrn,,l{^i)m,]]) ^ (13) 



where the average is over the patterns ^ in the overlaps m^i{(T''^), and 
are the aheady stationary overlaps in the previous layer. The sum over ^ in 
the exponent is over both, /i < /c (condensed) and fi > k (non-condensed) 
patterns. 

We follow the extension of the standard procedure to calculate 
{[Z"']c)patt and obtain the free energy per site in the large- A'^ limit by means 
of the method of steepest descent. We find the replica-symmetric form 

/ = ^a[Jo + /?r(l-g)] + ijo j;?n2 + ^{log[l-/3Jo(l-(?)] 
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l-/3Jo(l-(?)J 2[l-/?Jo(l-(/)]^^^ 

i( j Dz\og{2cosh(3[Y,C^^iJomf, + Jm^) + z^/^'^^^^ ^(14) 



ix<k 



where is given by Eq.(12), Dz = (27r)~2e~2^ dz and all quantities refer 
to any given layer, except for rh^ which refers to the previous layer. Here, 
m^, q and r are the replica-symmetric parameters that will be determined as 
solutions of the saddle-point equations df /dm^j, = df /dq = df /dr = 0. The 
condensed overlap for < k will be taken as an input from the previous 
layer and the sum over non-condensed patterns, X^^->;. provides a further 
link with that layer which will be determined below. 

The parameters are interpreted as follows: = (m^/(<T^)) is the 

replica symmetric average overlap with pattern for all replicas a given 
by Eq.(5), while q = (t'^o-'^) is the replica symmetric average of the spin- 
glass order parameter qafs = jjYli'^ia'^iis^ ^'^^ where the brackets 
stand for a thermal and configurational average over the patterns. As usual, 
r accounts for the introduction of the spin-glass order parameter into the 
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replicated partition function and it is given here by 

+ (^PJo- ^nr znz ' ^^^^ 

for a / /?, where Xa = Yl-yi^ ~ l^'^oQ]ay which / is the identity matrix 
and q = {qa/s}, while z G 3ft'" and Dz = (27r)""/2e~^^^dz. 

In order to determine X]^~>/j consider the formal relationship for the 
derivative of the free energy with respect to the control parameter Jq, which 
can be derived directly from Eqs.(8), (12) and (13) 



Y^{ml/a^))=a-a[3Jo^{l-q')-Y.{m%iicT'))-2^^ , (16) 



in replica symmetric form. Note that the second term on the right-hand side 
contains the noise in the form — /3(1 — (/^)A^/Jo that will be cancelled by a 
similar term from the last part of the equation. Using the explicit expression 
for the free energy in Eq.(14), one finds 

relating the non-condensed overlaps in two consecutive layers, where q is the 
spin-glass order parameter in layer /. A similar relationship can be derived 
for layer I — 1 where {'mf^i_2) becomes the stationary input tti^ for that 
layer. Using this relationship with the saddle-point equation df/dq = 
for layer / — 1, in order to eliminate X]^>fc("i^ ;_2)' applying again 
the saddle-point equation to layer I we obtain the third recursion relation 
below between the macroscopic order parameters in two consecutive layers 
{m,q,r) {m',q',r'). 

Thus, altogether, the saddle-point equations for the dense network in 
which c = 0(1) yield the recursion relations 

m' = (^$, j Dzi&nh [3\^$,.{Jom' + Jm) + z^J a[f-' + Jl{l - c)q'/c] (18) 
l' = {j Dzianh^ l3^i.{Jom' + Jm) + z^ a[r' + J2(l - c)q'/c] (19) 
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f'[l-/3Jo(l-gOf-JoV = /3V2(l-^)V-A + -4T^T^ , (20) 

i - pJo[^ - q) 

where the last equation is formally the same as that for the fully con- 
nected recurrent layers in terms of the rescaled variable f = r/c. Here, 
m = (mi, . . . , rrik) is the condensed overlap vector in one layer and m' = 
(m'l, . . . ,m'j^) in the next layer. Given a state of the first layer, Eqs.(18)- 
(20) describe the evolution of the states of the network. We recover the 
fixed-point equations for a diluted purely recurrent network when J = for 
Jq = 1 [2^ and the recursion relations for the layered feed-forward network, 
with full connection of units between layers, when Jq = for J = 1 |7j. For 
c = 1 we recover the equations of Coolen and Viana jllj . 

Note that the synaptic dilution in the case of the dense network enters 
only through the variance of the noise in the effective intra-layer interac- 
tions. In contrast, in the extremely dilute limit c — > 0, it is more appropriate 
to rescale (231121] " a' = a/c = p/cN, the ratio of stored patterns per 
mean number of connected sites, in place of rescaling r. Eq.(20) becomes 
then a recursion relation that depends explicitly on c. 

Following earlier work, we consider either a clamped input or a free 
relaxation in the first layer . In the first case, the state cr^ in that layer 
may be any randomly chosen configuration with a given condensed overlap 
m. The first two recursion relations remain the same and the last one is 
replaced by 

f'[l-f3Jo{l-q')f-Jiq' = j' . (21) 

On the other hand, in the case of a free relaxation the macroscopic state 
{m,q,r) of the first layer that follows from Eqs.(18)-(20) when J = is 
determined by the equations 



m 



^ y"z5ztanh/?{^.Jom + zy^^[fTj^(l^^^J)^})^ (22) 
j Dz tanh^ p[$.Jom + a[f + J'^{1 - c)q/c] (23) 



(24) 



We recover the known equations for the diluted recurrent network when 
Jo = 1 j21]- Thus, Eqs.(18)-(24) extend the model of Coolen and Viana 
for the case of symmetric synaptic dilution in the recurrent layers. We 
take next the limit T ^ and eliminate the parameter redundancy writing 
Jo = ^(1 +u;) and J = Ul-oj) [H]. 
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4 Phase transitions 



We turn next to the effects of dilution on the phase transitions in the model 
and consider three situations as in earlier work JJ. First, we look for 
the storage capacity of infinitely long chains in which a stationary state is 
reached. Then we focus attention on the first and the second layer in a 
search for multiple transitions between replica-symmetric states. The third 
transition we discuss is the replica-symmetry breaking transition at the de 
Almeida-Thouless (AT) line. 

4.1 Saturation transition in an infinitely long diluted chain 

We consider here the transition where the overlap disappears with an in- 
creasingly large storage of patterns. A stationary state is reached along the 
network when (m',q',r') = {m,q,r) and Eqs.(18)-(20) become 



m 



^ j Dz tanh /?|^ .m + a[r + J'^{1 - c)q/c] | (25) 

j L»ztanh2 (j^^.m + z^a\f ^ J2(l - c)g/c] (26) 
(1 - ujf + q(\ - 2qCuj{\ ^ uj) 



4[l-iC(l+u;)][l-C7(l+w)+c^C2] ' 



(27) 



where C = /3(1 — q). Taking the limit T — > and focusing attention on 
pure states, where = m5^x, for some A, we can do the integrations and 
the averages and follow the usual procedure [S] to reduce Eqs.(25)-(27) to a 
single equation, now in terms of the scaled overlap 



X = m/-^2a[f + J2(l-c)/c] , (28) 
where f is given by Eq. (27) and q ^ 1 in the limit T — > 0. We also have 

exp(-x^) (29) 



C 
and 



2 



lTTa[r + J^{l - c)/c] 



xya = 7- r Y 1 ("j'Jj 

[(1 + 1^2) er/2(x) B{x, a;) + i (1 + c^)2 A(x, w)] 2 
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where 



A{x, uj) 



[erf{x) 



exp(— x^)] [er/(a;) — 




{l + uj)x 



exp(— x^)] 



X 



[er/(x) 



(31) 



erf{x) 



(w + w^) 2x 



(32) 



(1 + 00^) A/vr 



When c = 1 one recovers the relationship for the fully connected dual model 
[TT] , and when uj = —1 {Jq = 0) or w = 1 (J = 0) one gets back the result 
for the layered network [7] or the result for the diluted recurrent network 
[2l| . respectively. 

The numerical solution of Eqs.(30)-(32) yields the critical storage ca- 
pacity ac{to,c), for a given u! and c, as the value of a where the solution 
with finite overlap (non-zero x) disappears. The result is shown in Fig. 2 
for various values of c. When uj = 1 (the purely recurrent network) one 
recovers the ac = 0.138 for the fully connected network with c = 1 [Hj. On 
the other hand, when uj = —1 (the purely layered network) ac = 0.269 for 
any value of c, since the dilution is only in the recurrent part [J]. There is 
a maximum value of ac ~ 0.317 for uj ~ —0.12, when c = 1, in consistency 
with a previous result and the maximum decreases and shifts towards 
the behavior of the layered network with increasing dilution. It has also 
been checked that although Oc decreases with increasing dilution, the ratio 
ac/c = 0.629, as one would expect in the extreme diluted limit c ^ for 

UJ=1. 

Although ac is a decreasing function of c for all values of w, as one 
would expect, the network still has a relatively large storage capacity for a 
moderate amount of dilution and a balanced ratio of interaction strengths. 
For instance, for c = 0.1 and uj ~ —0.20 (J/Jq ~ 1.5), ac is still the same 
as that for the fully connected network. Thus, the synaptic interactions 
between layers do not need to be much stronger than the interactions within 
the layers for a typical good performance with a moderately finite degree of 
dilution. As pointed out in the introduction, this may be of use for biological 
networks. 

4.2 Multiple phase transitions in the first layers 

We consider now the operation of the first two layers and we study the 
effects of dilution on multiple phase transitions that already appear at the 
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junction between the first and the second layer Even hmiting the study 
to pure states as we do here, with = ra5^^\ for some A, the overlap is 
layer dependent and we denote it by m and m' for the first and the second 
layer, respectively. 

We deal first with the clamped operation in the first layer, with a given 
overlap vector m, and take the T ^ limit in Eqs.(18),(19) to obtain 
m' = erf{y), where 

Jom' + Jm 



y 



V2a[f' + J^(l - c)/c] 



(33) 



is the appropriate scaled overlap and f' is given by Eq.(21). In that limit. 



q' 1 and C 



/3(1 
C'r. 



becomes 
2 



lTTa[f' + J^{1 



exp(-y^ 



'ov-^ - c)/c] 

When this is used we obtain the non-linear equation for y, 



G{y,a,m) F{y)+m 



1 



yyla 



1 + P(- 



1 



in which 



G^{y,a,m) 



erfiy) - 

/vr 



1 



.1 



c lerf{y)+m{-^ 



(34) 

(35) 

(36) 
(37) 



and p = 1. Solution of Eq.(35) with Eqs.(36) and (37) yields the overlap m' 
for the second layer and the phase diagrams presented below. 

We consider next the case of free relaxation of the first layer, and restrict 
ourselves again to pure states. First, Eqs.(22)-(24) have to be solved for the 
scaled overlap in the limit T ^ and when the solution is taken as an input 
in Eqs.(18)-(20), in the same limit, one obtains a new overlap m' on the 
second layer for this mode of operation. Thus, we find first m = erf{w), 
where 

w = Jom/J2a[r + J2(l - c)/c] (38) 



and C = P{1 — q) is now given by Eq.(29) with w in place of x and F{w) is 
given by the same expression as in Eq.(36). We find the equation 



G(w, a, Oj 
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that yields the scaled overlap w for free relaxation of the first layer. Following 
similar steps as for the clamped operation, with the full expression for f' in 
Eq.(20), in place of Eq.(21), we find formally the result given by Eq.(35) 
but now with 



When c = 1, G{y,a,m) = 1 for any m, and we recover the results for the 
fully connected network. 

It can be seen from the above equations that, as long as the dilution re- 
mains finite, there is a clear distinction between the two modes of operation, 
as in the fully connected network Already when m = 0, in the case of a 
clamped input with p = 1, Eqs.(35)-(37) yield a critical Qc for a bifurcation 
of a solution with y ^ (meaning m' ^ 0), for any finite c. In contrast, one 
finds that p = oo for free relaxation of the first layer and in that case the 
only solution is y = 0, that is, m' = 0. 

To obtain all the solutions that appear for general values of m (in the 
case of clamped input) and c, we look for the bifurcations from Eq.(35) that 
are given by this equation and its derivative with respect to y which have 
to be solved simultaneously. This can be done either for the clamped input, 
with p = 1, or for the case of a free relaxation of the first layer, with p given 
by Eq.(40). The outcome for the case of a clamped input is shown in Fig. 
3 for m = and in Fig. 4 for m = 1, in both cases for various values of c, 
together with the saturation transition for the infinite chain in each case, for 
reference. This is not the saturation transition for the chain of two layers 
we are discussing here, except for u = 1, as will be seen below. 

The phase diagram for the two-layer chain when m = has two regions. 
In region I, everywhere above the solid phase boundary m' = is the only 
solution. In region II, below the solid line, there are three stable solutions: 
m'l ~ 1, 1712 ~ —1 and rrig = 0, such that \m[\ / |?n.2l, except for w = 1 
where |m']^| = Img]. The phase boundary of multiple solutions of region II 
meets the saturation transition of the infinitely long chain at w = 1. This is 
the place where the model becomes a set of purely recurrent networks, with 
no interaction between layers and, hence, the stationary states of the second 
layer are already those of the infinite chain. The solution with overlap zero 
and q' = 1, above and below the phase boundary, describes a spin-glass 
state. 

In the case of an input m = 1.0, one obtains two further regions of 
coexisting stable states shown in Fig. 4. Now in region II one has two 
solutions, m'l ~ 1 and ~ —1, such that ^ |"^2l- region III 

there are two stable retrieval solutions with m[ ~ 1, m'2 smaller than one. 
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In region IV there are three stable retrieval solutions m'j^ ~ 1, m'2 ~ — 1 
and rrig small and positive, such that 7^ |?7i2l' except for uj = 1 where 
\m'i\ = \m'2\. The small solution m'^ vanishes precisely at a; = 1, leaving 
the known results for the purely recurrent network, regardless of dilution. 
The phase boundaries that go up to a; = 1 again meet there the saturation 
transition of the infinitely long chain. 

Again, all three regions of retrieval are reduced with increasing dilution, 
particularly regions III and IV, but they are still there for a finite dilution 
of c = 0.1. In the case of extreme dilution only region II is left over a tiny 
part of the phase diagram in terms of a = p/N. But in this limit it is 
more appropriate to consider a' = a/c, which remains finite. Similar results 
are obtained for other finite values of the fixed input overlap m. We also 
analyzed the effects of free relaxation of the first layer on the performance 
of the second layer and found qualitatively similar results to those described 
here for the case of clamped input. 

As one can see, the presence of these regions and the coexistence of 
various retrieval states is a feature of the competition between the recurrent 
and the layer information processing, which is considerably diminished with 
increasing dilution. When c = 1 we recover the earlier results for the fully 
connected network |llj . 

4.3 The replica- symmetry breaking (RSB) transition 

We look here for the effect of synaptic dilution on the de Almeida-Thouless 
(AT) transition j^T] where the replica symmetric solution for the saddle point 
equations ceases to be valid at the AT line. There is RSB at sufficiently low 
T in the fully recurrent network, that is when a; = 1, and there is no RSB 
in the purely layered network with a; = — 1. In both, the fully connected 
and the diluted dual model RSB appears as soon as w > —1, that is already 
for an arbitrarily small recurrent interaction Although the AT line is a 
boundary in the (T, a) plane and our work, as well as that in that ref. [11], 
are restricted to T = one may get an idea of the size of the RSB region by 
looking at the deformation of that boundary with a decrease of oj towards 
a; = — 1, and that is what we do next. 

The AT line in the present model is obtained considering the fluctuations 
around the replica symmetric forms for the spin glass order parameter qafj = 
{o'iaO'ip) and the auxiliary parameter qaf3 given by the right-hand-side of 
Eq.(15) for a ^ (3. Proceeding in the usual way on the free energy prior to 
the assumption of replica symmetry we obtain the AT line for the diluted 
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dual model, 



X cosh ''/3|^.(Jom + Jm) + z^J a[f + Jl{l - c)q/c] . (41) 

We recover the expression for the purely recurrent network when Jq = 1 
and J = |21] and that for the fully connected dual model when c = 1 jllj . 
The replica symmetric solution Eqs.(18)-(20) is stable when the right-hand 
side of Eq.(41) is less than one. This corresponds to the regions above the 
AT lines shown in Fig. 5 for three different values of oo when c = 1, c = 0.1 
and c = 0.001. It can be seen that the domains of RSB below the AT 
lines continue to become smaller with a decrease of lv (that is with reduced 
recurrent interactions) towards uj = —1 where the AT line coincides with 
the T = axis as in the fully connected network model The effect of 
(finite) dilution is to increase the region of RSB with a shift of the AT lines 
towards smaller values of a = p/N. Nevertheless, the effects of RSB are 
smaller than in the purely recurrent network justifying the analysis in this 
work based on the assumption of replica symmetry. 



5 Conclusions 

We studied in this work the effects of finite symmetric dilution on the per- 
formance of a dual model that combines information processing of recurrent 
and feed- forward networks. The model consists of a feed- forward chain of 
recurrent networks and the dilution is in the symmetric synaptic interactions 
of the recurrent layers. Our analysis extends the original work of Coolen 
and Viana with symmetric synaptic interactions between units within the 
layers and non-symmetric interactions between units in consecutive layers. 
The competition between these interactions is responsible for the behavior 
shown in the phase diagrams. On one hand, a small inter-layer feed-forward 
interaction between layers produces the multiple transitions that are found 
either in the case of clamped operation with finite m or in the case of free 
relaxation. In the other extreme, an infinitesimal symmetric recurrent in- 
teraction already yields a RSB transition. 

A stationary state replica analysis was carried out in this work assuming 
a frozen-in dilution of synaptic interactions in order to study the saturation 
transition in infinitely long chains and the multiple transitions that already 
appear at the junction between the first and second layer. We found that 
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finite synaptic dilution produces a gradual change of the performance of the 
model reducing all the regions of stable retrieval states, in particular those 
where multiple solutions appear due to the competition between recurrent 
and feed-forward processing. One would expect that the study of further 
initial layers should yield a performance closer to that of long chains. 

Although the performance of the dual model is reduced by synaptic 
dilution, we showed that there can still be a considerable output in a long 
chain of layers in the case of a finite amount of dilution of the synapses 
within the layers and pointed out that this feature could be relevant for 
dual models of biological networks. 

There are several extensions that one may conceive of the work presented 
in this paper. It would be interesting to consider the effects of non-linear 
synapses of a general form [22] and one could also think of relaxing the sym- 
metry of the synaptic interactions within the layers. A dynamical approach 
would be necessary in that case. 
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Figure 2: Critical storage capacity Oc for an infinitely long chain with 
various degrees of dilution, as indicated. The fully connected network, with 
c = 1, is also shown for reference. Increasing dilution is described by a 
decreasing value of c. 
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Figure 3: Phase diagram for the two-layer network under clamped operation 
with overlap m = in the first layer for various degrees of dilution, as 
indicated. The saturation transition (in dotted lines) is shown for reference 
and the phases I and II are described in the text. 
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Figure 4: Phase diagram for the two-layer network under clamped operation 
with overlap m = 1 in the first layer for various degrees of dilution, as 
indicated. The saturation transition (in dotted lines) is shown for reference 
and the phases I-IV are described in the text. 



21 




7.0x10""^ 1.4x10"^ 2.1x10"^ 

a 



Figure 5: The replica-symmetry breaking transition (AT line) for to = 
1.0, 0, —0.5 and various degrees of dilution. For lo = —1 (purely layered 
network) the AT line coincides with the T = line. 
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